Visual Basic Source Code

home *** CD-ROM | disk | FTP | other *** search

/ Visual Basic Source Code / Visual Basic Source Code.iso / vbsource / fcc110 / fcc.doc < prev next >

Wrap

Text File | 1993-10-11 | 28.4 KB | 743 lines

FCC Version 1.10 10/5/93 What is FCC ------------------------------------------------------------------------- FCC is a file access library which may be used in place of 'standard' file access functions. FCC performs the functions of a disk cache and disk doubler. FCC enhances standard file access functions in two ways: - FCC uses an efficient LRU write back cache. FCC supports cache block sizes as small as 128 bytes for increased performance when randomly accessing small blocks. - FCC can optionaly provide sequential and random read/write access to a compressed file. Like a disk doubler, FCC performs these functions transparently. All you our your application is aware of is that the physical size of the FCC file is smaller. Uses for FCC ------------------------------------------------------------------------- - To speedup file access. FCC is much faster than a disk cache, and the cache parameters are under program control. Programs which randomly access small blocks of information will benefit the most. - To reduce the disk requirements of applications. Both for distribution, and when installed. FCC may be used in place of the Windows LZ functions. - To hide sensitive data. Data compression ensures that proprietary data can not be easily decoded. - To update existing products. Many database type programs waste a lot of disk space. FCC can remove this waste with file compression, requiring only minor changes to the application. - To efficiently store flat file databases. With FCC, you do not need to be concerned about balancing field lengths with disk usage. Why use FCC ------------------------------------------------------------------------- FCC does not have any runtime royalties, and does not require a separate compression program (like COMPRESS.EXE). In addition, FCC provides full write access to compressed files. Programming Interface ------------------------------------------------------------------------- The following programming interface is not language specific. FCC may be used by almost all Windows languages and tools. For language specific interface information see the files: FCC.H FCC.BAS The FCC functions closely resemble the standard C file access functions (open, close, lseek, read, write). This is also the model used by many operating systems. An internal file pointer is kept for each opened file. It is used to direct the file positioning used in read and write operations. In the standard model there are 7 functions: open - open or create a file for reading and/or writing close - close an opened file seek - set the internal file pointer to a specific file location read - read from a file write - write to a file tell - retrieve the position of the internal file pointer length - retrieve the length of a file The open function returns a handle to the opened file, which is used to identify the file to the other functions. FCC also has 5 additional functions: version - determine the version of FCC being used exists - determine whether a file exists and/or is compressed plength - retrieve physical length of a compressed file flush - write dirty cache blocks to disk copy - copy one file to another The main difference between the FCC model and the standard model is in file opening. The additions to the standard open function are: - A way to indicate the cache size. - A way to indicate the cache block size. - A way to indicate whether or not a created file should be a compressed file. The FCC functions are presented below: FCC_open ---------------------------------------------------------------------- PURPOSE: Open or create (and open) a file. PARAMETERS: handle - Filled in with a FCC file handle (if no error). name - Name of file to open or create. flags - Open flags. May be one of: FCC_READ - Open for reading. FCC_WRITE - Open for reading and writing. FCC_CREATE - Create and open for reading and writing. compress - Compression method (only used with FCC_CREATE). May be one of: FCC_CDEFAULT - Default compression. FCC_CNONE - No compression. kbytes - The size of the file cache in kilobytes (1 - 2048). blcksize - The size of cache blocks in bytes (128 - 16384). RETURN: 0 - No error. else - An FCC error code. NOTES: The blksize used when creating a compressed file will be used in all subsequent access to the file. To change the blksize, copy the file with a different block size. When a file is opened, the internal file pointer is set to 0. FCC_close ---------------------------------------------------------------------- PURPOSE: Close an opened FCC file. PARAMETERS: handle - FCC file handle. RETURN: 0 - No error. else - An FCC error code. NOTES: All dirty cache buffers will be written to disk before the file is closed. This may take some time if the cache used is very large. FCC_seek ---------------------------------------------------------------------- PURPOSE: Set the internal file pointer to a specific location. PARAMETERS: handle - FCC file handle. pos - File position offset. where - May be one of the following: FCC_SEEKSET - Seek to position. FCC_SEEKCUR - Seek from current position. FCC_SEEKEND - Seek from end of file. newpos - Filled in with the new internal file pointer position. RETURN: 0 - No error. else - An FCC error code. NOTES: The internal file pointer is 0 based (i.e. the first byte in a file is located at position 0.) To position the file pointer for appending, use FCC_seek(handle, 0, FCC_SEEKEND, NULL). If newpos is set to NULL, it will not be filled in. FCC_read ---------------------------------------------------------------------- PURPOSE: Read from a file. PARAMETERS: handle - FCC file handle. buff - Memory buffer where data should be placed. bytes - Bytes to read (0-FFFE). rbytes - Filled in with number of bytes read (if no error). RETURN: 0 - No error. else - An FCC error code. NOTES: rbytes may be filled in with a value which is less than the actual number of bytes requested. FCC_write ---------------------------------------------------------------------- PURPOSE: Write to a file. PARAMETERS: handle - FCC file handle. buff - Memory buffer containing data to write. bytes - Bytes to write (0-FFFE). wbytes - Filled in with number of bytes written (if no error). RETURN: 0 - No error. else - An FCC error code. NOTES: wbytes may be filled in with a value which is less than the actual number of bytes requested. FCC_tell ---------------------------------------------------------------------- PURPOSE: Retrieve the current internal file pointer position. PARAMETERS: handle - FCC file handle. pos - Filled in with the current file pointer position (if no error). RETURN: 0 - No error. else - An FCC error code. NOTES: The value placed in pos is 0-based relative to the beginning of the file. If the file pointer is positioned at the first byte of the file, pos contains 0. FCC_tell functions identically to: FCC_seek(handle, 0, SEEK_CUR, &pos); FCC_length FCC_plength ---------------------------------------------------------------------- PURPOSE: Retrieve the logical (FCC_length) or physical (FCC_plength) length of a file. PARAMETERS: handle - FCC file handle. length - Filled in with the file length (if no error). RETURN: 0 - No error. else - An FCC error code. NOTES: If the file is not compressed, FCC_plength functions identically to FCC_length. If the file is compressed, FCC_plength will typically return a value smaller than FCC_length. FCC_exists ---------------------------------------------------------------------- PURPOSE: Determine if a file exists and/or is compressed. In addition, FCC_exists may be used to determine a files length. PARAMETERS: name - Name of file. version - Filled in with: -1 - The file does not exits. 0 - The file exists but is not FCC compressed. else - The FCC version used to compress (if no error). length - If not NULL, filled in with the logical file length. plength - If not NULL, filled in with the physical file length. RETURN: 0 - No error. else - An FCC error code. NOTES: The value placed in version will be the integer equivalent of the FCC version used to compress. Version 1.00 will be 100, version 10.15 will be 1015. All versions numbers will be between 100 and 9999. FCC_version ---------------------------------------------------------------------- PURPOSE: Determine the FCC_version PARAMETERS: none RETURN: The version of the FCC library. The version is the same as described in FCC_exists. FCC_copy ---------------------------------------------------------------------- PURPOSE: Copy one file to another. The source file may be any file, including FCC compressed files. The destination file may optionaly be compressed. PARAMETERS: src - Source file. dst - Destination file (or NULL, or ""). blksize - Block size parameter to both FCC_open calls. Choose the largest block size for maximum compression, and the fastest copy. compress - Indicates whether the destination file should be compressed. May be one of: FCC_CDEFAULT - Compress destination file. FCC_CNONE - Do not compress destination file. progress - See section Copy Progress below. RETURN: 0 - No error. else - An FCC error code. NOTES: The destination file will have the same date and time of the source file (if no error). If dst is NULL or "", the file will be copied in place. In other words, src will be copied to itself. - To decompress a file: FCC_copy(filename, "", blksize, FC_CNONE, NULL) - To compress or de-fragment a file: FCC_copy(filename, "", blksize, FCC_CDEFAULT, NULL) Copy Progress ------------------------------------------------------------------------- This section is only relevant to the function FCC_copy. When copying a file using FCC_copy, you may want your application to be able to abort the copy and/or display progress information to the operator. The progress parameter to FCC_copy is used for this purpose. If you do not want any progress information, you may pass a NULL parameter for progress. In MS Visual Basic this is equivalent to passing ByVal 0&. The structure FCC_COPYT is used for progress/abort purposes. The fields in FCC_COPYT are: wHnd - window handle message - message to be sent to wHnd wparam - first message parameter lparam - second message parameter cancel - if not zero, copy will be aborted length - logical length of input file plength - physical length of input file curpos - bytes copied so far Periodically, FCC_copy will send the following Windows message: SendMessage(wHnd, message, wparam, lparam) If, after sending this message, the cancel field of progress has been set to a value other than 0 by the application, FCC_copy will return with the 'error' FCC_ERR_CANCELED. In addition, the application may use the values in length, plength and curpos to display progress information. The MS Visual Basic example demonstrates how the progress parameter to FCC_copy is used. There are several important notes about the VB example: - The message sent in this example is WM_KEYUP with wparam set to 256. In other words, a false key up message is sent to the main VB Window. The false key is 256, which will not conflict with any Windows keystrokes. - DoEvents is called to process any user input. If DoEvents is not called, Windows will not be able to process any user input or display progress information. - When the false WM_KEYUP message is sent, progress information is not automatically displayed. The display of progress information is done by a timer control. This is done to reduce CPU overhead and for an 'aestheticly pleasing' once/per half second display. FCC Error Codes ------------------------------------------------------------------------- FCC_ERR_OPEN Could not open a file. This error will typically be encountered when trying to open a file which does not exist. FCC_ERR_WRITE Could not write to a file. This error will typically be encountered if a disk drive fails or becomes full. FCC_ERR_READ Could not read from a file. This error will typically be encountered if a disk drive fails. FCC_ERR_CLOSE A file could not be closed. This error will typically be encountered if a disk drive fails. FCC_ERR_SEEK A seek could not be performed. Because seek operations should not generate an error, this error is very unlikely. FCC_ERR_CACCESS Data in a compressed file could not be retrieved. This error will typically be encountered if a FCC compressed file has become corrupt or has been altered without FCC. FCC_ERR_VERSION A file could not be opened because it is recognized as a compressed FCC file, but the version is not supported. FCC_ERR_NOMEM Insufficient memory. FCC_ERR_PARAM An invalid parameter was passed to a FCC function. This error will typically only be encountered when a program is being written. Make sure each of the passed parameters is valid. FCC_ERR_CANCELED The cancel field of progress in FCC_copy was set to a value other than 0. FCC_ERR_UNKNOWN An unknown error occurred. Disabling the Write Back Cache ------------------------------------------------------------------------- There is no way to explicitly disable the write back cache. You may, however, write a function which does so. In C: int FCC_writeflush(int handle, void *buff, long bytes, long *wbytes) { int err; err = FCC_write(handle, buff, bytes, wbytes); if (err == 0) err = FCC_flush(handle); else FCC_flush(handle); return (err); } This function writes and flushes in one operation. Choosing Cache and Block Sizes ------------------------------------------------------------------------- When opening a file with FCC, you have to tell FCC how much memory to use for the cache, and what cache block size to use. The cache will have (kbytes * 1024) / blksize blocks. In other words, if you specify a 1MB (1024K) cache and a 256 byte block size, there will be 4096 cache blocks. Each cache block can contain blksize bytes of file data. A least recently used (LRU) block allocation mechanism is used. In general, you should make the cache size as big as possible (up to the size of the file of course). Choosing the block size is more difficult. The following rules will usually work in most cases: - When accessing a file sequentially, choose a block size as big as possible (up to the size of the cache size). - When accessing a file randomly, choose a block size between 1/2 and 4 times the size of the typical access size (the bytes parameter to FCC_read and FCC_write). The other consideration when choosing a block size is compression. The larger the block size, the smaller the file will be. The penalty, however, is slower random access. For example, suppose you randomly access a compressed file, and the typical access size is 128 bytes. File access using a block size of 32K will be much slower than with a block size of 2K. A block size of less than 1K for compressed files is not recommended. It is best to experiment with different combinations until an adequate size vs. speed tradeoff is reached. Zeroing Unused Data ------------------------------------------------------------------------- When writing to a compressed file, all unused space should be set to one value (zero is suggested). For example: Suppose you are writing an address book database. For each record you are reserving 30 bytes for the last name. Since most last names have less than 30 letters, a lot of space will be unused in each record. By setting this unused space to one value, it may be compressed by FCC. The last name 'Jones' should be stored as 'Jones' followed by 25 0's. Fragmentation ------------------------------------------------------------------------- When randomly accessing a compressed file, it may become fragmented. This causes the physical file size to become larger than it needs to be. By simply copying the FCC file, the physical size will be become as small as possible. For example: Suppose you have been randomly accessing the file MY.DAT, and now you want to make it as small as possible. FCC_copy may be used for this purpose: FCC_copy("MY.DAT", "", blksize, FCC_CDEFAULT, NULL) Other Versions / Other Products ------------------------------------------------------------------------- There are other versions and products which do not have a standard pricing plan: - Other versions of FCC with faster and/or better data compression. - Buffer based data compression algorithms. - Stream based data compression algorithms. - MS-DOS and OS/2 libraries. Ordering Information ------------------------------------------------------------------------- To purchase FCC 1.10, print (or copy) this ordering form. You will receive the commercial version of FCC 1.10, and the right to distribute FCCWIN.DLL for use with your applications. The price shown is an introductory price, valid through 12/31/93. After this date, call for the current pricing. Product ID: FCC110 Price: $95 US (non-US shipping add $5 US) Make Payment To: Four Lakes Computing Payment Methods: US Check, US Money Order, US Dollars, American Express International Money Order. Send Payment To: Four Lakes Computing 1135 Williamson #4 Madison, WI 53703 USA Address: __________________________________________________ __________________________________________________ __________________________________________________ __________________________________________________ Phone: _____________________ Disk: ___ 5 1/4 ___ 3 1/2 If you have any questions, contact: US Phone: 608-256-3382 Compuserve ID: 70662,2501 The Test Program FCCTST.EXE ------------------------------------------------------------------------- FCCTST.EXE is a Windows program which may be used to test FCC. It is written in Microsoft Visual BASIC. The source code for FCCTST.EXE is contained in the files: FCC.BAS - FCC BASIC Interface FCCERR.BAS - Converts FCC error codes to English messages. FCCTST.FRM - Visual BASIC form. FCCTST.MAK - Visual BASIC project file. FCCTST.EXE allows you to copy files. The source file may be any file, including FCC compressed files. You may choose for the destination file to be compressed. FCCTST.EXE also allows you to test random read and write access to both compressed and uncompressed files. The controls in FCCTST.EXE are explained below: Source File: ---------------------------------------------------------------------- This is the source file for a copy operation. The file you specify will NOT be modified by FCCTST.EXE. The choice of Source File is not really important if compression is not being used (any file of the desired size will work). If compression is being used, Source File should contain the type of data you are interested in compressing. Destination File: ---------------------------------------------------------------------- This is the destination file for the copy operation, and the file that will be modified during random access. THIS FILE MAY BE CREATED OR MODIFIED BY FCCTST.EXE. The default destination file is c:\fcc.tmp. Copy ---------------------------------------------------------------------- Copies the source file to the destination file. Random ---------------------------------------------------------------------- Randomly accesses the destination file until cancel is pressed. Cancel ---------------------------------------------------------------------- Cancels a file copy or random access. Cache Size (K): ---------------------------------------------------------------------- Used to determine the cache size for the destination file. Block Size ---------------------------------------------------------------------- Used to determine the block size for FCC_copy and FCC_open. When accessing a compressed file, it is suggested that block size be no less than 1024. For maximum compression, choose the maximum Block Size for the destination file (16384). Compress ---------------------------------------------------------------------- Determines whether the destination file in a copy operation should be compressed. Random Size ---------------------------------------------------------------------- Indicates the size of reads (and possibly writes) used in a random access operation. A random number between 1 and Random Size will be used for each access. Data will be read from the destination file, and if Random Writes is enabled, will be written to another location in the destination file. This parameter is important for testing cache efficiency during random access. If the average size of access you want to test is 1024 bytes, choose a Random Size of 2048. Random Writes ---------------------------------------------------------------------- Indicates that random writes (as well as reads) should be performed in a random access operation. It is interesting to note that the physical size of a compressed file will increase for awhile when accessed randomly with Random Writes enabled. This is because of file fragmentation. Random % ---------------------------------------------------------------------- Indicates the location of random access: 10 - 90% of the random access requests will be made to the first 10% of the file. The other 10% will come from the entire file. 30 - 70% of the random access requests will be made to the first 30% of the file. The other 30% will come from the entire file. 100 - The file is accessed randomly. This is an important parameter for testing the effectiveness of the cache. As most files are not accessed truly randomly, this parameter allows you to test typical file access, and adjust the cache parameters for the best performance. Progress Information ---------------------------------------------------------------------- Source File Size Indicates how big the input file is. The logical size will be in parentheses if it differs from the physical size. Destination File Size Indicates how big the destination file is. Note that this may grow during randow access (with writes enabled) due to fragmentation. Bytes Processed Indicates how many bytes have been copied, or how many bytes have been processed in a random access operation. K/Sec Indicates how may kilobytes have been processed per second. This is the main indicator to you of cache/compression efficiency. It will typically range between 5 and 2000. This is the gauge you use to select optimal FCC parameters.